In the present context, noise is broadly defined as random variations in value among the pixels, that comprise a digitally represented image (an “image”), from an ideal representation of the subject of the image. The images can be real or synthetic. Many images include such noise as a result of the processes used to acquire the image from a real-world source or introduced during subsequent processing or reproduction. In the fields of photography, graphic design, visual special effects, production of printed materials, creation of composite photographs, film and video editing, encoding and transmission of images or film or video, and other related fields, it is desirable to control the exact noise content, in both type and amount, of the images being dealt with. For example, excessive noise is often unwanted, in that it may detract from perfect reproduction or interfere with certain subsequent technical processes. Conversely, it may be desirable to add noise to an existing image for aesthetic purposes or to allow for seamless composition with other noisy images.
Noise includes such effects as analog video dropouts, so-called “salt and pepper” noise where occasional isolated pixels are randomly changed to full or zero intensity, dust and scratches on photographic or motion picture negatives or prints, and other image imperfections known generally as “impulse noise”. The major characteristic of impulse noise is that it affects only a few isolated and usually randomly located pixels of an image. Impulse noise is generally different to another category of noise called “broad-band”.
Generally, broad-band noise alters all pixels of the image. Examples of such noise include film grain, noise originating in the light-sensitive component (e.g. CCD array) of digital still or video cameras, and analog static and in general noise caused by electrical fluctuations in the components of an electronic image recording or reproduction system.
Although not strictly random other types of image effects, present in all pixels of an image are halftone patterns or dither patterns used for image reproduction, or artifacts arising from the image encoding process, specifically so-called “compression artifacts” resulting from digital image or video compression algorithms.
Often, it is desired to remove noise from an image or image sequence. This could be for aesthetic reasons or as in the field of digital image and video compression prior to storage or transmission, image noise of almost any sort reduces the efficacy of such compression systems, because it represents spurious “detail” that the compression algorithm attempts to reproduce. Almost always, the exact noise pattern of an image—as opposed to the general characteristics of the noise—is not of interest, so encoding this information wastes valuable storage space or transmission time.
Noise reduction in digital (and analog) images is a classic problem in signal processing and has been studied for some time. Prior art includes many different types of filtering systems. All of these have various shortcomings. Some filters do not operate well on color images or on image sequences. For example, an algorithm not specifically designed to process image sequences may be temporally unstable with the result that changing artifacts are introduced on a frame-to-frame basis causing a popping or strobing effect which is known as image “chatter”. Shortcoming of almost all image noise reduction systems is the amount of noise that can be removed without adversely affecting the image content. All noise reduction systems degrade the source image to some degree during processing, taking the form of a loss of fine image detail or an overall blurring of the image. This tradeoff is unavoidable as fine detail is difficult to distinguish from noise, however a good algorithm will minimize the amount of degradation induced at a given level of noise reduction.
Thus, reducing noise while preserving image sharpness and detail is a difficult problem. Current systems do not operate effectively both across the color channels (if present) and between the frames (if present) of an image sequence. Even for still or monochrome images, existing algorithms represent an often poor solution for broad-band noise, resulting in either little noise reduction or excessive blurring. Hence, the noise reduction techniques employed in practice on broad-band noise are often limited to mild Gaussian blurs or the application of a median filter. On the other hand, adding noise to an existing image is a simpler problem. Precise control over the exact spatial, chromatic, and temporal structure of the noise is desirable. Previous systems have been fairly successfully in achieving this goal, but often the desired effect is not noise in and of itself but matching of the noise structure of two different images. For example, in the field of motion picture special effects, often a computer-generated object is inserted into a photographed scene. The scene is stored as a digital image and contains film grain noise as a result of its chemical photography origin. The image of the synthetic object does not, so if these two images are composited naively the lack of film grain where the computer-generated object is visible detracts from the realism of the resulting scene. This defect can be remedied by applying noise which exactly matches the film grain of the real scene to the synthetic image before compositing.
More generally, images obtained at different times, captured with different equipment or recording media, or even captured using the same system but subsequently processed differently, must often be combined into a single composite image. For example, multiple photographic elements recorded under identical conditions will have mismatched grain noise if they appear scaled differently the final composite image. Simply extracting the noise from one image, replicating it, and re-applying it to the other images making up the composite will not solve this problem, because in this case each image already has pre-existing noise that will interact cumulatively with any newly applied noise.
Accordingly, there is a need for a system targeted specifically to reduce broad-band noise which is capable of exploiting all available information, using multiple frames and color channels if available.
Further there is a need for a system that is capable of automatically analyzing noise in one image and for processing a second image match the noise between the two images, regardless of pre-existing noise in the second image.